On the reversibility of the observed process of 
three-state hidden Markov model 



CHEN Yong 

School of Mathematics and Computing Science, Hunan University of Science and Technology, 
Xiangtan, Hunan, 411201, P.R.China, chenyong77@gmail.com 



Abstract 

For the continuous-time and the discrete-time three-state hidden Markov model, 
the flux of the likelihood function up to 3-dimcnsion of the observed process is 
shown explicitly. As an application, the sufficient and necessary condition of 
the reversibility of the observed process is shown. 
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1 Introduction 

There has been a large amount of literature published on the time reversibil- 
ity in probability, such as References [TJ[2l[3l|4l[5j[6j[71[8], which are mainly 
about the Markov processes and the semi-Markov processes (or Markov renewal 
processes). In the case of a Markov process with finite state (discrete time or 
continuous time), Kolmogorov's criterion for time reversibility is well-known. In 
the References [9J [TUl EJ dH El , they examined time reversibility in the context 
of a univariate stationary linear time series (Gaussian or non-Gaussian) and of 
multivariate linear processes. 

In Reference [13], for the hidden Markov model, it is shown that the re- 
versibility of the observed process is not equivalent to that of the underlying 
Markov chain, i.e., if the underlying Markov chain is reversible, then the ob- 
served process is reversible too, however, if the Markov chain is irreversible, 
then the observed process is either reversible or irreversible. In Reference [15j . 
the necessary and sufficient conditions for reversibility of hidden Markov chains 
on general (countable) spaces are obtained, however, the reversibility therein is 
concerning the complete process, i.e., the bivariate stochastic process containing 
both the underlying process and the observed process. That is to say, the above 
two types of reversibility of the hidden Makov model are different completely. 

In the present paper, for continuous-time three-state Markov processes, we 
calculate the flux of the likelihood function (joint probability distribution) of the 
observed process. As an application, the sufficient and necessary condition of 
the reversibility of the three-state hidden Markov model is shown (in the sense 
of [14], not in the sense of [E]). In fact, besides the reversibility of the underly- 
ing Markov process, the reversibility of the observed process is distinguished by 
whether the state-dependent probability matrix is regular (Definition 13.21 Thc- 
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orem l3.4]) . We illustrate by an example that one cannot detect irreversibility in 
some cases by comparing directional moments like that used in |14[ pi 04]. 

Wc have also investigated the discrete-time three-state hidden Markov model. 
Since the method is similar to the continuous-time case, we list the conclusions 
in Appendix (Section 4) and omit most of the proofs. For the discrete-time 
case, the reversibility is also related to whether zero is an eigenvalue of the 1- 
step transition probability matrix (Proposition I4.6[) . Here we see a difference 
between discrete-time and continuous-time hidden Markov model. 

The reversibility of the hidden Markov model may be of interest in some 
biological studies. An approach to modelling the DNA sequence is to use a hid- 
den Markov model; see, for example, Reference [16j[T7j. Since DNA sequences 
have directions, we should rule out the reversible hidden Markov model. 



2 The flux of the likelihood function 

Let {St : t £ R + } be the observed process with state space S = {0, 1, 2, • • ■ , K — 1}. 
Definition 2.1. The ?i-dimension likelihood function of {St : t G R + } is defined 



as Pr(S' tl = si, 5 t2 = s 2l ■ ■ ■ , S t 



where nefj and ^ t\ ^ t 2 ^ • • • ^ 



t n . The flux of the likelihood function of {St ■ t 6 K + } is defined as 
Pr(S , tl = si, S t2 =s 2 , ■■■ , S tn = s n ) - Pr(S t - = Si, S t - = s 2 , ■ ■ ■ , S t - = s n ), (1) 

where t^ = t\ + t n — tk- 

Let {Ct : t £ M. + } be an irreducible three-state Markov process with the 
transition rate matrix Q, and the stationary distribution /i = (jxi, fi 2 , M3), where 
Mi +M2 +M3 = 1, fii > 0. 



Q = 



-a\ a 2 a 3 
bi -b 2 63 
ci c 2 -c 3 



(2) 



where a\ — a 2 + a 3 , b 2 = bi + b 3 , c 3 = Ci + c 2 , a;, bi, a > 0, i = 1,2, 3, and 
016203, fei + Ci, a 2 + C2, 03 + 63 > (i.e. irreducible). By the stationarity, it is 
clear that the transition rate flux of {Ct ■ t £ E + } is 



Mi«2 - A^i = M263 - M3C2 = M3C1 - ^ia 3 . 
Let v = fiia 2 — \i 2 b\. And the eigen-equation of Q is 

A(A 2 +aA + /3) =0. 



(3) 



(4) 



4/3. 



Denote by — Ai, — A 2 the nonzero eigenvalues of Q. Let A = or 

Similar to the denotation of Reference [1.8;, let S\ and Tf denote the se- 
quence from 1 to j of the observed states and observation times. The Markov 
assumption for the hidden process is given by 



Pr[Cfe) I C(h), C(t 2 ), . • • , Cfe-i), St 1 , T> 
= Pr[C(t j )\C(t j - 1 ),Ti_ 1 =ti_ 1 ] 



(5) 
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where the quantity P Cj - 1 ,c j ('tj ~ tj-i) denotes the transition probability of oc- 
cupying sate Cj at time Tj = tj given that the process was in state Cj-\ at tj—%. 
As indicated by the last equality, the transition probabilities of the process are 
assumed to be time homogeneous. We also assume that, conditional on the 
state of the hidden process at time tj , an observation Sj is independent of all 
previous observations and the hidden process prior to time tj: 

Pv[Sj | C(ti), C(t 2 ), . . . , C{tj), St\T{ = tj] 

= Fv[S j \C(t j ),T j = t j ] (6) 

Let the 'state-dependent probability' (i.e., emission probability, signal prob- 
ability) matrix be n = (?r(fc | i)), i = 1, 2, 3; fe = 1, 2, . . . , A" - 1 (i.e., a 3 x A 
matrix) . Note that the rows of 11 must sum to 1 L 1 ]. Let ip k be the fc-column of 
n and A fc = diag {<p k } , k = 0, 1, 2, . . . , A - 1. 

Proposition 2.2. 77ie /lure of the 2-dimension likelihood function is when t > 0, 



Pr {So =i,S t = j} - Pr {So = j, S t = i} = 



vA 



Ai — A2 



r e -A 2 t _ g-Aiti 



where v is the transition rate flux, A = (j/2 — a^X^i — 21) — (#2 — 22X2/1 — 
(xi,2/i,zi)' = and (x 2 , y%, Z 2 )' = tfj. 

Corollary 2.3. // the rank of the state- dependent probability is 1 or 2, then 
Pr {So =i,S t = ]} - Pr {So = j, S t = 1} = 0. 

Theorem 2.4. The flux of the following 3-dimension likelihood function is when 
r,t>0, 



Pr {So = S r = S r+t =i}~ Pr {S = S t = S t+r = i} 

— vD C e -A 2 r-Aii _ g-A 2 t-Air\ 

where v is the transition rate flux, D = [x — y)(y — z)(z — x), (x, y, z)' = <pi 



Proofs of Proposition 12.21 Corollary 12.31 Theorem [23] arc presented in Sub- 
section 12.11 

2.1 Proofs 

Let U = diag {/Lti, //2, A«3}. Then 



UQ-Q'U = ^ 



1 -1 
-1 1 

1 -1 



(7) 



Let e = (1, 1, 1)' and the matrix L = e\i. 

Lemma 2.5. //A^O, then for t 6 M + , the t-step transition probability matrix 
is 

P(t)=gtL + dtQ + f t l, (8) 



1 The state-dependent probability here is the transpose matrix of that in Reference 1141 . 
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where 

, _ e -A 2 t_ e -A lf 
<k - — 173^ . 

, _ X 1 e-^-X 2 e-^ t ( 9 ) 
Jt — A1-A2 ' V ' 

9t = 1 - ft- 

It is Proposition 4.3 of Reference [19]. The reader can also refer to Theo- 
rem 14.9 of Reference [50]. To write it in terms of function with matrix coeffi- 
cients is the key to the results in the present paper. 

Remark 1. Fix the value of Ai, and let A2 — > Ai, then one has 

d t = t e~ Xlt 

f t = (l + x' 1 t)e- x ' t , (10) 
9t = 1 - ft- 

It is exactly the t-step transition probability matrix in the case A = 0, please 
refer to [TOl [20] and the references therein. That is to say, Eq. ([T0|) is the same 
as Eq.(|0J in the sense of limit. We therefore do not distinguish whether A = 
or not for all the subsequent formulas. 

Proof of Proposition \2.2\ Since L = efi, we have 

fiAiLAje = (^A ! :e)(/iA_ ) e*) = (/lA^eX/zA^e) = ^AjLA,;e. 

Note that A{Aj = AjAi. By Eq.(2.27) of Reference Q3] and Lemma [231 we have 

Pr{S*o = i,S t = j} - Pr{5 = j,S t = i} 
= fiAiP(t)Aje - fiAjP(t)Aie 

= (iAi(g t L + d t Q + f t \)Aje- + d t Q + / f l)A,e 

= g t (fiAiLAje- fiA 3 LA. t e) + d t {^A l QA j e - /iA.,QA;e) + f t {^A l A :j e - fiA 3 A z e) 
= dtbiAiQAje - fiAjQAie). 

Since [lAiQAje — tp^UQcpj, we have 

jitAjQAje = (ffjUQifi = (^UQ^)' = f'iQ'^fj- 
By Eq.©, we have 

fiAiQAje- fj,AjQAie = (f' t (UQ - Q'U)<Pj = vA. (11) 
This ends the proof. □ 

Proof of Corollary POI Note that A = det{H}, where "det" is the determi- 
nant function, and H = [e, ipi, <pj] is a 3 x 3 matrix. If the rank of the state- 
dependent probability is 1, then ipi, ipj are linear dependent and we obtain that 
A = 0. If the rank of the state-dependent probability is 2, and if ipi, ipj are 
linear independent, then they are one base of {(fik,k = 0, 1, 2, . . . , A" — 1}. Note 
that e = J2k=o Pi- Then ip^, cpj, e are linear dependent. Thus A = 0. This 
ends the proof. □ 



Proof of Theorem \2.4\ Since L = e/z, we have 

^AjLAiQAje = (fiAie)(fxAiQAie) = (jxAiQAie)(jj,Aie) = /iA^A.LA.e. 
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Similar to Proposition ^. 2[ we obtain 

Pr{5 = S r = S r +t =i}~ Pt{Sq = S t = S t+r = i] 

= fiAiP{r)AiP{t)Aie - MA i P(i)A i P(r)A i e 

= /uAj(c/ r L + d r Q + f r \)Ai(gtl + d t Q + f t l)A,e 

- LiAtigtL + dtQ + f t \)A % (g r L + d r Q + f r \)A l e 

= {g r d t - g t d r )(^A t lA t QA t e - /iA. t QA. t LA. t e) + (g r f t - Sf t / r )(/iA i LA?e - ^A- LA, 

+ (d r / t -d f / r )(MA l QA J 2 e- i uA2QA i e) 

= (d r ft - d t / r )(MiQA-e - MiQAie). 

It follows from Lemma |2"31 that 



drft - d t fr = —^—{e-^-^ - e- A2t - Air ]- 

Ai — A 2 

Let ?/> = (x 2 ,y 2 ,z 2 y. Similar to Eq. fTTj) . we have 
/zAiQAfe- fiAfQAte 

= ^(UQ-Q'U)V 

= i/[a;(y 2 - z 2 ) + y(z 2 - x 2 ) + z(a; 2 - y 2 )] 1 ' 

= vD. 

This ends the proof. □ 



3 The reversibility of the observed process 

The observed process is the same as in Section [21 

Definition 3.1. The observed process is said to be reversible if its finite- 
dimensional distributions are invariant under reversal of time, i.e., the flux of 
the likelihood function vanishes, 

Pr(S tl = si, S t2 =82, ••■ , S tn = 8 n )-Pi(S t - = si, S t - = s 2 , ■ ■ ■ , S t - = s n ) = 0, 

where = t\ + t n — tk, for all positive integers n and all ^ t\ ^ t 2 ^ ■ ■ ■ ^ t n . 

Definition 3.2. Two rows of a matrix are said to be equal if they are two 
equal vectors. If any two rows of the state-dependent probability matrix l~l are 
not equal, we say that l~l is regular. Otherwise, we say that l~l is singular, i.e., 
there are at least two undistinguishable states among the three hidden states 
by means of observation. 

Theorem 3.3. If the underlying Markov process is reversible, then the observed 
process is reversible too. 

Although the proof in Reference [HJ P102-103] is about the discrete-time 
hidden Markov model, it is still valid for the continuous-time one and is ignored 
here. 

Theorem 3.4. The observed process of the continuous-time three-state hidden 
Makov model is irreversible, if and only if the underlying Markov process is 
irreversible and the state- dependent probability matrix is regular. 
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Proof of Theorem 13.41 is presented in Section 13.21 

If n is regular, the rank of n is 3 or 2. If 11 is singular, the rank of 11 is 2 or 
1. That is to say, the rank of 11 is involved in the reversibility of the observed 
processes. 

By Reference [21j . the hard limiting (or clipping) transformation is very 
useful from a practical viewpoint, and the rhythm inherited in the binary series 
carries a great deal of information about the original series. If we maintain 
the regularity condition of 11 when clipping the observed process of the hidden 
Markov model, it preserves the time-reversibility property by the last theorem 
(e.g., Example [2]). 

The time reversibility of high-order hidden Markov models (e.g., more than 
four-state) is difficult to be solved completely. We can only find some simple 
sufficient conditions of the irreversibility, for example, the underlying Markov 
process is irreversible and the rank of the state-dependent probability matrix 
is equal to the number of states of the underlying Markov process (similar to 
Proposition 13. 5[) . 

Remark 2. Let the complete process be {X t = (C t , St), t ^ 0}. Clearly, it is 
still be a finite-state Markov process, please refer to [35]. Similar to Theorem 2.1 
of Reference [TS], by Kolmogorov's criterion, we can show that the complete 
process is reversible if and only if the underlying process is reversible. That is 
to say, there are two different types of reversibility of the hidden Makov model. 



3.1 Applications 



Example 1. The deterministic function of a Markov process is a special case of 
hidden Markov model. Let / = (/i, f^, fzj' be a function defined on the state 
space. If / = (1, 1, 0)' or / = (1, 0, 0)' like that used in Reference [23], then the 
state-dependent probability matrices are respectively 



rii 



o 1 

1 

1 



1 

1 
1 



Since the state-dependent probability matrices are singular, the observed pro- 
cesses is reversible by Theorem [ 



Example 2. Suppose {Ct, t 0} be the irreversible Markov process with tran 
sition rate matrix 

" -2/3 1/3 1/3 
2/3 -1 1/3 
1/2 1/2 -1 

Let {StjEI, {£t} j {Vt} be three observed processes with state-dependent proba- 
bility matrices respectively 



Q = 



(13) 





" i 










" i 










" i 





rii = 


1/4 


1/2 


1/4 


, n 2 = 


1/4 


1/2 


1/4 


, n 3 = 


1/4 


3/4 










1 




_ 1/2 


1/3 


1/6 







1 



Since all the state-dependent probability matrices are regular, the observed pro- 
cesses are irreversible by Theorem 13.41 {ry t } is clipped from {St} and preserves 



2 {St} comes from the example in Reference 1141 p!05]. 
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irreversible. Since both the rank of ll 2 and II3 are 2, by Corollary 12.31 we have 
that 

Pr {6 = i, 6+r = j} = Pr {6 = 3, 6+r = i} , 
Pr {?7 t = i, r) t+r = j} = Pr {r) t = j, Vt+r = i} ■ 

In the case, one cannot detect irreversibility by comparing directional moments 
E (6£l+r) and E (£T6+r) with n G N like that used in Reference QH pl04]. 

3.2 Proof 

Let ei = (l,0,0)',e 2 - (0,1,0)', e 3 - (0,0,1)', E, = diag{ej, A fc = 
diag{v? fe }. 

Proposition 3.5. If the underlying Markov process is irreversible and the rank 
of the state- dependent probability is 3, then the observed process is irreversible. 

Proof. Since the rank of the state-dependent probability is 3, we can choose 
a base of R 3 , without loss generality, to be <po,ipi,ip2- Then e\ = Ei=o ^fii 

^2 = Ei=0 VWh and E l = Ei=0 A i X i> E 2 = Ej=0 NVr 

Suppose on the contrary that the observed process is reversible. When t > 0, 

= PT{S = i,S t =j}-Pv{So = j,S t = i} 

= fj,AiP(t)Aje - ^A J -P(t)A i e, where i,j = 0,1,2. 

Thus 

MiPia(*)-A*2P2i(t) = ME 1 P(t)E 2 e-/iE 2 P(t)E 1 e 

2 2 
= 2J XiyjnAiP(t)Aje- ^ y J x i ^,A j P{t)A l e 

i,j=0 i,j=0 
2 

= X] Xiy^AiPAje- [MAjPAie] 

i,j=0 

= 



Note that 



lim = Q. 



Then ^ = /zia 2 — [iib\ = 0. However, the underlying Markov process is irre- 
versible, i.e., v ^ 0, a contradiction. □ 

Lemma 3.6. If the rank o/TI is 2, £/ien its columns {ifi, i = 0, 1, 2, . . . , K — 1} 
are t/ie linear combination of its certain column and e = (1, 1, 1)'. 

Proof. If the rank is 2, then Jv ^ 2. Without loss generality, let {ipi, </? 2 } be 
one base of {(fii, i = 0, 1, 2, . . . , K — 1} with ^ ce, where c£l Note that 
e = Eilo 1 V« = + ^ 2 > w here y 7^ 0, .t G R, thus <^ 2 = (e — xipi)/y. Since 
{Vii V2} is one base, all {<^i, i = 0, 1, 2, . . . , K — 1} are the linear combination 
of {<£>!, e}. □ 

Without loss generality, if the rank of n is 2, let i = 0, 1, 2, . . . , K — 1} 
be the linear combination of ipi and e. Let A = diag {vi}- 
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Proposition 3.7. Suppose that the underlying Markov process is irreversible. 
If the rank of V\ is 2, and if any two rows of 11 are not equal, then the observed 
process is irreversible. 

Proof. We claim that (p\ = (x, y, z)' with x ^= y ^ z. Without loss of generality, 
suppose on the contrary that ip\ = [x,x,z)', then the first two rows of 11 are 
equal by Lemma 13.61 a contradiction. Thus 

D = (x-y)(y~z)(z-x)^0. 

Since the underlying Markov process is irreversible, the transition rate flux 
v 7^ 0. By Theorem 12.41 when r, t > and r ^ t, 

Pr {So = S r = S r+t = 1} - Pr {S Q = S t = S t+r = 1} 0, 

i.e., the observed process is irreversible. □ 

Proposition 3.8. If there are two equal rows of 11, then the observed process 
is time reversible. 

Proof. If the rank of 11 is 1, i.e., all the three rows of 11 are equal, the observed 
process is in fact identical independent distribution series. 

If the rank of 11 is 2, without loss of generality, suppose the first and the 
second row of 11 are equal. Then tp Sk = Xke + yue 3 and A Sk = X}~\ + y/cE 3 . The 
flux of the likelihood function is 

P r (<Sti = Si, Sij+tj, = S2, ■ • ■ , Stj+tj,-! |-t r = s r ) 

— Pr(/Stj = S r , S't 1 +t r = S r -1, ■ ■ • , Sti+tr-i ht2 = s i) 

= H A Sl P(h) As 2 P{t 3 ) ■ ■ ■ P(U) As r e- n A Sr P(t r ) A Sr i P(i r _i) • • • P{t 2 ) A Sj e 
= fi[xi\ + yi E 3 }P(t 2 )[x 2 \ + 2/ 2 E 3 ]P(t3) • • • P(t r )[x r \ + 2/rE 3 ]e 

- ^[avl + y r E 3 }P(t r )[x r ^i\ + j/ r _iE 3 ]P(t r _i) • • • P(t 2 )[xi\ + yiE 3 ]e. 

Expand the expression, and delete the terms which do not contain E 3 . Note 
that Eg = E 3 for / € N(i.e., E 3 is projective matrix), P(t)e = e, fiP(t) = fi, and 
P(t)P(r) = P(t + r) for t, r £ M + . All other terms pairwise satisfy that 

fi ■ ■ ■ P(U)\ ■ ■ ■ P(t 3 )E 3 ■ ■ ■ S- n- ■ ■ E 3 P(t 3 ) ■ ■ ■ \P(U) • • • e 
= ^(E 3 P(r)E 3 ) ■ • • (E 3 P(i)E 3 )e- ^(E 3 P(t)E 3 ) • ■ • (E 3 P(r)E 3 )e 
= MaPsaW • • • P 33 (i) - /i 3 P 33 (t) ■ ■ • P 33 (r) 
= 0. 

This ends the proof. □ 
Proposition 3.9. Suppose that the underlying Markov process is irreversible. 

1) If the state- dependent probability matrix 11 is singular, then the observed 
process is reversible. 

2) If the state- dependent probability matrix 11 is regular, then the observed 
process is irreversible. 

Proof. The first case is Proposition ^. 81 If 11 is regular, then the rank of 11 is 3 
or 2. Thus the second case is Proposition 13 . 71 and Proposition ^. 51 □ 

Proof of Theorem \3.4\ It can be shown directly by Theorem 13.31 and Proposi- 
tion EU □ 
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4 Appendix: the discrete-time case 

Let {St : t £ Z + } be the observed process with state space S = {0, 1, 2, ■ • ■ , K 
Let {Ct : t € 7L + { be an irreducible three-state Markov chain with the 1-step 
transition probability matrix P and the stationary distribution /i = /x 2 , /U3), 
where /ii + /i 2 + M3 = lj Mi > 0. 



1 — 02 — 03 a 2 a 3 
61 1 - 61 - b 3 b 3 

ci c 2 1 - ci - c 2 



(14) 



By the stationarity, it is clear that the probability flux is 

[1x0,2 - n 2 bi = pL 2 h - /Lt 3 c 2 = M3C1 - yUia 3 . (15) 

Let v = \i\di — /U 2 &i. Let Q = P — I, where I is the unit matrix. Denote by 
— Ai, — A 2 the nonzero eigenvalues of Q. 

Lemma 4.1. I/A^O, then for n e N, the n-step transition probability matrix 
is 

P" = g n l + d n Q + f n \, (16) 

where 

, _ (l-A 2 )"-(l-A 1 )" 

"» ~~ A1-A2 

f _ Ai(l-A 2 )"-A 2 (l-Ai)" 

Jn — A1-A2 ' V 7 

,9 n = 1 - /«• 

Proposition 4.2. T/ie /ilux 0/ i/ie 2-dimension likelihood function is when n S 
N, 

7/ /J 

Pr {S = i, S n = 3} - Pr {S = j, S n = 1} = -[(1 - A 2 )" - (1 - Ai)™], 

where v is the probability flux, A = (y 2 — %2)(%i — z\) — (x 2 — z 2 )(yi — x{), 
{x\,y\,zi)' = ifi, and (x 2 , y 2 , z 2 )' = ipj. 

Theorem 4.3. The flux of the following 3-dimension likelihood function is when 
n, m S N, 



Pr {So — S n — Sn+m — t} — Pr {5*0 — S m — S m + n — i} 
= AT^K 1 - W 1 - A i) m - (1 - - A 2 )™], 

where v is the probability flux, D = (x — y){y — z){z — x), (x,y,z)' = ipi- 



Theorem 4.4. The three-state hidden Makov models is irreversible, if and only 
if the underlying Markov chain is irreversible, the state- dependent probability 
matrix is regular, and one of the following conditions is satisfied: 

1 ) the rank of V\ is 3, 

2) the rank of V\ is 2, and is not the eigenvalue of P. 

Proofs of Lemma 14.11 Proposition 14.21 and Theorem 14.31 are omitted. Proof 
of Theorem 14. 41 is presented in Subsection l4.ll 
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4.1 Proofs 

Lemma 4.5. If the rank of n is 2, the observed process is reversible if and only 
if for all n,m, . . . ,k £N, 

/iAP n AP m A • • • AP fc Ae - ^iAP k A • • • AP m AP"Ae = 0, (18) 

where A = diagj^i}. 

Proof. The necessity is trivial. We need to prove the sufficiency only. It follows 
that ip Sk = x k e + ykfi from Lcmma l3.6l Then A Sfc = XkS + j/feA. The flux of the 
likelihood function is 

Pr(5i = si, £2 = S2, • ■ • , Si = si) - Pr{Si = Si, = s 2 , • • ■ , 5"i = s;) 
= /i A Si P A S2 P • • • A S; e - /i A s P A s _ P • • • A Sj e 
= n[ Xl \ + yi A]P[x 2 \ + y 2 A]P ■ • • [xi\ + yiA]e 

- n[xi\ + yiA]P[xi-!\ + yi-iA]P ■ ■■[x 1 \ + yi A]e 

= ^2 x i± ■■ ■x is y jl ■■ -y jk [n- ■ -P\- ■ -PA- ■ -e- /j,- ■■ AP ■■ -\P ■■ -e\. 

{ii,i 2 ,--- ,i s } 

where {Ji,i 2 ,--- ,is} e {1,2, ••■ , 1} and j 2 , ■ ■ ■ ,jk} = {1,2, •■■ , Z}\{«i, • • • ,i s 
Delete the term which does not contain A. Note that fiP = /i and Pe~ e. 

jU ...p|...p A ...e-//---AP---IP---e 

= /iAP™AP m A • • • AP fc Ae - f iAP k A ■ ■ ■ AP m AP"Ae 
= 0. 

This ends the proof. □ 

Remark 3. Ea. (fT8| is equivalent to for all positive integers r and all ^ t\ ^ 

t 2 < ••• ^t r , 

Pr(S tl =S t2 = --- = S tr = l) = Pr(S t - = S t - = ■ ■ ■ = S t - = 1), (19) 
where = t\ + t r — t\. 

Proposition 4.6. // the rank of the state- dependent probability is 2, and is 
the eigenvalue of the 1-step transition probability, then the observed process is 
reversible. 

Proof. Without loss generality, let 1 — X 2 = 0. By Lemma |2~51 for all n G N, 

P n = d n P + g n L, 
where g n = 1 - d n , d n = (1 - Ai)"" 1 . 

/iAP"AP m A • • • AP fc Ae - M AP fc A • • • AP m AP"Ae 
= /it A [d n P + g n L] A [d m P + g m L] A • • • A [d k P + g k L] Ae 

- (J.A [d k P + 5fe L] A • • • A [d m P + g m L] A [d n P + g n L] Ae 

= V] d n ■ ■ ■ d ls g jl ■ ■ ■ g jk [//A • ■ • PA ■ • • LA • • • Ae - /iA ■ • • AL ■ • • AP • ■ • A 

{tl,t2,-" 
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Delete the term which does not contain L. Note that L = e/i. 



fj,A ■ ■ ■ PA • • • LA • • • Ae 

OAP • • • PAe)(/iAP • • • PAe) • • • (^iAP • • • PAe) 
/LtA---AL---AP--- Ae. 



This ends the proof by Lemma 14.51 



□ 



Proposition 4.7. Suppose that the underlying Markov chain is irreversible. 
Then we have 

1) if there are two equal rows of U, then the observed process is reversible; 

2) if the rank of V\ is 2, 

a) and if is the eigenvalue of P, then the observed process is reversible; 

b) n is regular, and if is not the eigenvalue of P, then the observed 
process is irreversible; 

3) if the rank of V\ is 3 ; then the observed process is irreversible. 

Proof. The first case is similar to Proposition 13.81 The second case is Propo- 
sition Hjni and similar to Proposition 13.71 The third case is similar to Proposi- 



Proof of Theorem \4-4\ It can be shown directly by Theorem 13.31 and Proposi- 
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